Skip to content

feat(docs): add docs sync system (#106)#107

Merged
mikkisguy merged 2 commits into
mainfrom
issue/106-docs-sync
Jun 22, 2026
Merged

feat(docs): add docs sync system (#106)#107
mikkisguy merged 2 commits into
mainfrom
issue/106-docs-sync

Conversation

@mikkisguy

Copy link
Copy Markdown
Owner

Closes #106

What changed

Adds a docs sync system that keeps docs/ in sync with the ShadowBrain database, so docs are browsable, searchable, and available to chat from the web UI.

  • src/lib/docs-sync.ts (new) — syncDocsDirectory engine: recursively upserts every .md file under the sync root as a note with source='docs-sync'. Idempotent (skip-unchanged by default), prunes rows whose files were deleted from disk, and supports --force (re-write all) and --dry-run (preview, zero writes).
  • scripts/sync-docs.ts (new) — CLI entry point mirroring import-markdown.ts.
  • src/lib/__tests__/docs-sync.test.ts (new) — 27 tests.
  • package.json — added sync:docs script; version 0.8.00.9.0 (minor: new user-facing capability).
  • README.md — version badge bumped in lockstep.

Sync behavior

  • Title = filename stem (getting-started.mdgetting-started).
  • Content = full raw markdown (frontmatter preserved) so docs are browsable verbatim.
  • Metadata = { "file_path": "docs/<rel>" }.
  • Stable id docs-sync-<sha256[:32]> — distinct prefix from notes (note-md-) so docs and notes sharing a relative path can't collide.
  • Each doc tagged #project:shadowbrain + #docs + a category tag: top-level files → #docs:<stem> (getting-started.md#docs:getting-started); nested files → #docs:<top-dir> (api/endpoints/auth.md#docs:api). Tags self-heal via INSERT OR IGNORE even on skipped files.
  • Prune is scoped strictly to source='docs-sync'; the prune audit log records the file_path (the id alone is an opaque hash).

Usage:

pnpm sync:docs              # sync docs/ → database
pnpm sync:docs --force      # re-write everything
pnpm sync:docs --dry-run    # preview changes

How verified

  • pnpm verify green: lint ✓, typecheck ✓, build ✓, pnpm test --run 719/719 ✓ (27 in the new docs-sync suite), knip ✓.
  • Smoke test against the real docs/: 36 docs discovered → 36 created; re-run → 36 skipped (idempotent); category tags verified correct across nested and top-level paths.

@oracle review

Two-pass strategic + security review (triggered: diff touches the database layer). No MUST-FIX blockers. All 5 SHOULD-FIX findings addressed before opening:

  1. Version bump is minor (0.9.0) per AGENTS.md §Versioning.
  2. Prune audit log now records the deleted file's path.
  3. Dropped a wasteful parseMarkdownFile call (title computed inline; avoids spurious frontmatter warnings).
  4. Added the hidden/private re-sync regression test (issue Security: two-level visibility (is_hidden + is_private) #54 scenario).
  5. Documented the cross-root footgun in the generateDocsId docstring.

Plus 3 nits addressed (self-heal test asserts tag identity, softened two overclaiming docstrings).

Intentionally deferred (out of scope for this PR)

  • No audit_logs row for skipped / too-large files — parity with the existing markdown importer, which also writes none.
  • walkMarkdownFiles / isHiddenPath duplicated from the markdown importer — extracting a shared module is unrelated churn.
  • ensureTag is not race-free against concurrent syncs — this is a single-operator CLI with no concurrent writers.

…106)

Adds a CLI tool (pnpm sync:docs) that recursively syncs every .md file
under docs/ into content_items as a note with source='docs-sync', tagged
#project:shadowbrain, #docs, and a path-derived category tag.

- src/lib/docs-sync.ts: syncDocsDirectory engine (idempotent upsert,
  tag self-healing, prune of deleted files, --force/--dry-run)
- scripts/sync-docs.ts: CLI entry point mirroring import-markdown.ts
- 27 tests covering idempotency, tags, cleanup, dry-run, force, and the
  hidden/private re-sync regression
- version 0.8.0 -> 0.9.0 (new user-facing capability)
@mikkisguy mikkisguy merged commit 459364e into main Jun 22, 2026
3 checks passed
@mikkisguy mikkisguy deleted the issue/106-docs-sync branch June 22, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add docs sync system: keep docs/ in sync with ShadowBrain database

1 participant